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Please commence national stage examination pursuant to 35 U.S.C. 371 for the 
above-identified PCT patent application. 

Enclosed are: 

[X] A copy of the international application as filed, including: 

[X] Abstract, 8 pages of specification and Claims 1-8. 

[X] Two (2) sheets of drawings showing Figures 1-4. 
[x] An English Translation of the application as filed, including: 

[x] Abstract, 6 pages of specifications and Claims 1-8. 

[x] Two (2) sheets of drawings showing Figures 1 -4. 
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[x] An English translation of amended page 2 of the specifications, 

[x] An English translation of amended Claims 1-8. 

[x] A Preliminary Amendment canceling Claims 3-8 and adding Claims 9-27. 

[x] An International Search Report along with an English Translation 
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If neither International Search fee nor International 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re Patent Application of Robert Frischolz 
Serial No.: Unassigned 
Filed: Herewith 

International Application No.: PCT/EP9907334 

International Filing Date: October 4, 1999 

Title: PERSONAL RECOGNITION METHOD AND SYSTEM, 
INCLUDING MODEL-BASED FINDING 



The Honorable Commissioner of 
Patent and Trademarks 
Washington, D.C. 20231 

Sir: 

Preliminary to the examination of the subject application, please amend the subject 
application as follows: 
In the Claims: 

Please cancel Claims 3-8 without prejudice. 

Please add the following new Claims 9-27: 

~ 9. (New) The method as claimed in Claim 1, wherein the binarized image first 
is compared on a small scale with a face model of corresponding small size, the area of 
the binarized image in which a face was found is enlarged and compared once again with 
a face model of corresponding larger size, the enlarging and comparing of the binarized 



PRELIMINARY AMENDMENT 



image area and face are repeated, as the case may be,^|l the face in the binarized 
image was localized with sufficient accuracy. 

--10. (New) The method as claimed in Claim 2, wherein the binarized image first 
is compared on a small scale with a face model of corresponding small size, the area of 
the binarized image in which a face was found is enlarged and compared once again with 
a face model of corresponding larger size, the enlarging and comparing of the binarized 
image area and face model are repeated, as the case may be, until the face in the binarized 
image was localized with sufficient accuracy. 

-11. (New) The method as claimed in Claim 9, wherein different face models 
with different resolutions are used depending on the size of the binarized image. 

- 12. (New) The method as claimed in Claim 10, wherein different face models 
with different resolutions are used depending on the size of the binarized image. 

-13. (New) The method as claimed in Claim 9, wherein the edge extraction for 
deriving the binarized image from the original image is carried out with different 
resolutions depending on the size of the binarized image. 

- 14. (New) The method as claimed in Claim 10, wherein the edge extraction for 
deriving the binarized image from the original image is carried out with different 
resolutions depending on the size of the binarized image. 

-15. (New) The method as claimed in Claim 11, wherein the edge extraction for 
deriving the binarized image from the original image is carried out with different 
resolutions depending on the size of the binarized image. 



-- 16. 



k (New) 




lethod as claimed in Claim 12, 



|in the edge extraction for 



deriving the binarized image from the original image is carried out with different 
resolutions depending on the size of the binarized image. 

- 17. (New) The method as claimed in Claim 9, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

-18. (New) The method as claimed in Claim 10, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

- 19. (New) The method as claimed in Claim 1 1, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

- 20. (New) The method as claimed in Claim 12, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

- 21. (New) The method as claimed in Claim 13, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

- 22. (New) The method as claimed in Claim 14, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 
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- 23. 



i. (New) 



tethod as claimed in Claim 15, wj 



kin the face model and/or 



the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

~ 24. (New) The method as claimed in Claim 16, wherein the face model and/or 
the binarized image are rotated and different steps of the rotation are used in depending 
on the size of the binarized image. 

- 25. (New) The method as claimed in Claim 1, wherein in the Hausdorff 
measure is determined on the basis of the average value of a certain percentage of the 
smallest minimum Hausdorff spacings, the percentage being between 0% and 100 %. 

~ 26. (New) A system for implementing the method as claimed in Claim 1, 
comprising a computing device for calculating the Hausdorff spacing and the Hausdorff 
measure on the basis of the dots of the binarized image and the face model. 

— 27. (New) A system for implementing the method as claimed in Claim 25, 
comprising a computing device for calculating the Hausdorff spacing and the Hausdorff 
measure on the basis of the dots of the binarized image and the face model. 



Respectfully submitted, 




L. Lawton Rogers, III 



Washington, D.C. 20005 
Telephone: (202) 898-1515 
Telecopier: (202) 898-1521 



1401 Eye Street, N.W., Suite 300 



Reg. No. 24,302 
D. Joseph English 
Reg. No. 42,514 
Mark C. Comtois 
Reg. No. 46,285 



Patrick D. McPherson 
Reg. No. 46,255 



Dated: April 4, 2001 
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Personal rec ognitio n method and system, 



including model-based face finding. 



The invention relates to a personal recognition method and system, including model-based 
face finding. 

There are methods and systems of biometric personal recognition based on a combination of 
face recognition, speech recognition, and lip movement recognition. The critical part thereof 
is the face finding. The invention relates to a method and a system of localizing the face of a 
person in any camera pictures within such a system of biometric personal recognition. 

The fundamental element in face recognition is the face finding, i.e. the precise localizing of 
the image sector which corresponds to the human face. Some solutions proposed in the prior 
art simplify these problems by requiring a uniform background in front of which the human 
face can be found. A face also can be recognized by including moving picture information, it 
being assumed that the face is contained only in that area of a picture which changes in the 
course of a picture sequence. 

Such simple approaches no longer meet present day requirements of face finding and face 
recognition. Personal recognition nowadays is assisted, for instance, by PC desk top cameras 
and other mobile cameras and, therefore, face finding must be robust and work with any kind 
of background, even with a moving background picture. 

The most successful approaches known to date in this field make use of neuron networks. 
These networks are trained with a great number of examples of faces. In this training an even 
greater number of "non-face images" are used as a counter-class so that, in the end, the neuron 
network will be able to distinguish between face and background. It is a disadvantage of this 
method, apart from the long computing time, that it is highly dependent on variants, like 
scaling, rotation, and distortion. 



It is, therefore, an object of the invention to indicate a novel system and method of face find- 
ing which is both robust and able to accomplish the face finding in real time. 
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This object is met, according to the invention, by a method comprising the features recited in 
claim 1 and a system comprising the features recited in claim 8. 

The method and system according to the invention are model-based. What this means is that a 
binary image of a human face or a so-called "line drawing" is searched for in a corresponding 
overall image. Thus an original picture first is converted into a binary image, for instance, by 
means of edge extraction. This binarized image then is compared with a binary face model to 
seek and find the image of the human face in the corresponding binarized total image. In con- 
trast to the known approaches, therefore, it is not the intensity value of the individual camera 
pixels which is compared or pixel variations which are looked for. Rather, the invention relies 
on a model structure of the type of a line drawing which may be available, for example, in the 
form of a bit image. 

The model of the face or the "line drawing" is compared with the binarized image by means 
of a modified version of the so-called Hausdorff spacing in numerous scaling, rotation, and 
distortion variants of the image. The application of the Hausdorff spacing to face recognition 
was described, e.g. by B. Takacs and H. Wechsler in "Face Recognition Using Binary Image 
Metrics", 2nd International Conference on Automatic Face and Gesture Recognition, IEFF 
Proceedings, April 1998. Reference is expressly made to that publication and the explanation 
it contains of the Hausdorff spacing. 

The publication mentioned describes the use of the Hausdorff spacing for purposes of face 
recognition. It is stated expressly that face finding is achieved by a method which is totally 
different. It was not taken into consideration in the prior art to apply the Hausdorff spacing for 
the purpose of face finding, among other things because this algorithm takes an awful lot of 
calculating time. 

The fundamental differences between the problems of face finding and face recognition 
should be recalled: Once the face is found in a picture, especially in a moving picture this 
facial picture can be compared with a collection of faces from a data base by almost any 
method of recognition or identification. If the face in question is contained in the data base the 
hit rates in general usually are very high, amounting to approximately 99 %. What is difficult 
with personal recognition, however, is the preceding step, namely first to find the face in any 
random picture and make a perfect "cutout" thereof. What may appear to be a minor differ- 



ence in a literal comparison of face recognition and face finding ultimately is decisive for the 
quality of the result of facial and, therefore, personal recognition. 

Image comparison by resorting to the Hausdorff spacing is based on the following principles: 
Two groups of dots are formed of the binarized image and the face model 

A = {«!,...,«„} 

and 

B ={&!,- K) 

The Hausdorff spacing then is defined by 

H(A,B) = max(h(A,B), h(B,A) 

with 

h{A,B) =maxmin||a — 611. 



The system and method according to the invention are not susceptible to the most frequent 
disturbing effects, such as rotation, different scaling, or distortion of the picture because it is 
easy to take these into account when comparing the dot groups. Nor is a long learning process 
needed for application of the method according to the invention, in contrast to the use of neu- 
ron networks. Other than with the approaches by way of neuron networks, furthermore, it is 
not necessary to predetermine, learn, or otherwise allow for any "non-face images". The sys- 
tem recognizes a face like a person does, based on the characteristics of the face itself rather 
than by relying on background traits which thus need not be taken into consideration. Special 
characteristics looked for (e.g. a person wearing glasses) quickly can be taken into account in 
the model by designing the model or "line drawing" accordingly. 

In the future, the system and method according to the invention may be used in biometric 
identification systems for automatic biometric access control with which often facial re- 
cognition is combined with speech recognition, lip movement recognition, retina scanning etc. 
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The invention permits to find the exact face sector, locate the exact eye position for retina 
scanning, locate the exact mouth position for calculating lip movements, and the like. 

But the system and method according to the invention can be applied much more universally 
too, for instance,yby establishing certain binary models so that only persons with certain fa- 
cial characteristics will be recognized in order to make a distinction according to such aspects 
as mimics, race, or gender. The method and system according to the invention are not even 
restricted to face finding because the model for which the search is made also might comprise 
a hand or another one or more parts of the human body or some matter. 

In its preferred embodiments the invention provides for use of a modified Hausdorff spacing 
to perform the face finding so that the calculating expenditure may be minimized and, there- 
fore, a result be obtained within reasonable computing time. The present embodiment of the 
modified Hausdorff spacing for face finding permits detection of approximately one face im- 
age per second. 

In addition, the invention provides a novel preselection of the image by special utilization of 
the voronoi surface, as further contribution to accelerating the process. 

The system and method according to the invention can be realized by software as well as 
hardware modules, the modified algorithms either being programmed or implemented as 
separate hardware so that they are real-time suitable, at least in the hardware embodiment. 

The invention will be described in greater detail below with reference to the drawings, in 
which: 

Fig. 1 shows an original picture taken with a digital camera; 
Fig. 2 shows a binarized version of the original picture of fig. 1 ; 
Fig. 3 shows a binary face model in the form a line drawing; and 

Fig. 4 shows the position of the face model found in the binarized image of fig. 2 with the 
assistance of the face model of fig. 3. 

Fig. 1 shows the original picture of a person, including the face, shoulder part, and back- 
ground. By edge extraction, this picture is converted into the binarized image illustrated in 



fig. 2. To accomplish that, the edges at the transitions between bright and dark in the original 
picture are used to prepare a kind of line drawing of the binarized image as illustrated in fig. 
2. What is looked for is the two-dimensional face model of fig. 3, with the aid of the Haus- 
dorff spacing, under the conditions explained in greater detail below. 

The general Hausdorff spacing offers a means of determining the similarity of one dot group 
with another one by examining the proportion of dots in the one group which are located close 
to dots in the other group, or vice versa. There are two parameters to decide whether the two 
dot groups are similar or not: (i) the maximum distance by which the dots may be spaced 
from one another and yet be observed as lying close to one another, and (ii) which is the 
maximum proportion of dots in one group that has this distance from the dots in the other 
group. 

Face finding by the measure of the Hausdorff spacing differs from other techniques, such as 
binary correlation, because there is no pairing of dots in the two groups which are compared. 
An explanation of the mathematical principles of the Hausdorff spacing may be found in the 
internet under the following address: http://www.cs.cornell.edu/ Vi- 

sion/hausdorffThausmatch.html. The principles of the Hausdorff spacing to which reference is 
made are explained in that document. 

The two-dimensional image of the picture presented in fig. 3 thus serves as a face model to be 
localized in the binarized image of fig. 2, making use of suitable two-dimensional transforms 
and scalings. 

Fig. 4 shows the best conformity of the model of fig. 3 with the binarized image of fig. 2 and, 
therefore, the position found of the model in the binary image in the sense that in fig. 4 the 
greatest proportion of binarized edge points of fig. 2 lie in the vicinity of the image dots of 
fig. 3. When applying the Hausdorff spacing, conformity between the face model and the bi- 
narized image is found even if the corresponding binary dots do not coincide precisely. 

To practice this model-based face finding method, the following modifications of the Haus- 
dorff spacing are made in a preferred embodiment of the invention in order to achieve face 
finding in real time: 
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The invention uses a hierarchic approach with which, to begin with, the binarized image is 
greatly reduced in size, a correspondingly small face model is searched for in the reduced 
binarized image; when a specific region is recognized as a likely location of the face looked 
for then this region and its surroundings are enlarged in order to continue with the search op- 
eration based on a model face of corresponding greater size, and so on. 

Different models are used in this hierarchic type of search, i.e. with the greatly reduced bi- 
narized image, for example, a model (fig. 3) including shoulders is used so that the re- 
cognition of the person will be reliable. The greater the scale of the binarized image, the finer 
the resolution and the more detailed the model will be for the face finding so that, ultimately, 
a model can be used which includes nothing but eyes, nose, and/or mouth, for example. 

The edge extraction for preparing the binarized image of fig. 2 likewise may be adapted to the 
respective hierarchic levels at which resolutions of different degrees of fineness are required. 
The invention provides for use of an adaptive Sobel filter to accomplish that. 

Respective suitable rotations of the image and/or model may be made within the various hier- 
archic levels. 

Additionally, the invention preferably provides for prefiltering of the binarized image (ero- 
sion, dilation, etc.). 

The modification of the Hausdorff spacing is another important feature of the invention. An 
especially preferred embodiment of the invention provides for working with a modified Haus- 
dorff spacing in which not only the average distance of all the minimum distances between 
the model and the image is utilized as the measure of the spacing. Instead, the average value 
of the first x % (0<x<100) of all the minimum distances serves as the basis for calculating the 
Hausdorff measure so that larger deviations ("escapes") will not be taken into account and 
falsify the result. 
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Claims 

1 . A method of finding a face in a binarized image by comparing the dot group of the 
binarized image with the dot group of a face model, wherein the dot groups of the bi- 
narizedyimage and of the face model are compared on the basis of the Hausdorff spacing be- 
tween the dots of the dot groups, and a face in the binarized image is recognized when a 
measure derived from the Hausdorff spacing fails to reach a limit value. 

2. The method as claimed in claim 1 , wherein the binarized image is derived from the 
original image by means of edge extraction. 

3. The method as claimed in claim 1 or 2, wherein the binarized image first is compared 
on a small scale with a face model of corresponding small size, the area of the binarized im- 
age in which a face was found is enlarged and compared once again with a face model of cor- 
responding larger size, the enlarging and comparing of the binarized image area and face 
model are repeated, as the case may be, until the face in the binazired image was localized 
with sufficient accuracy. 

4. The method as claimed in claim 3, wherein different face models with different reso- 
lutions are used depending on the size of the binarized image. 

5. The method as claimed in claim 3 or 4, wherein the edge extraction for deriving the 
binarized image from the original image is carried out with different resolutions depending on 
the size of the binarized image. 

6. The method as claimed in any one of claims 3 to 5, wherein different steps of rotation 
are used depending on the size of the binarized image. 

7. The method as claimed in any one of the preceding claims, wherein the Hausdorff 
measure is determined on the basis of the average value of the smallest x % of all minimum 
Hausdorff spacings, with 0<x<100. 
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8. A system for implementing the method as claimed in any one of the preceding claims, 
comprising a computing device for calculating the Hausdorff spacing and the Hausdorff 
measure on the basis of the dots of the binarized image and the face model. 
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Abstract of the Disclosure 

Personal recognition method and system, 
including model-based face finding. 



The invention relates to a face finding method and system in a binarized image by comparing 
the dot group of the binarized image with the dot group of a face model, wherein the dot 
groups of the binarized image and of the face model are compared on the basis of the Haus- 
dorff spacing between the dots of the dot groups and a face is recognized in the binarized im- 
age when a measure derived from the Hausdorff spacing fails to reach a limit value. 
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Fig. 3 



Fig. 4 
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This object is met, according to the invention, by a method comprising the features recited in 
claim 1 and a system comprising the features recited in claim 8. 

The method and system according to the invention are model-based. What this means is that a 
binary image of a human face or a so-called "line drawing" is searched for in a corresponding 
overall image. Thus an original picture first is converted into a binary image, for instance, by 
means of edge extraction. This binarized image then is compared with a binary face model to 
seek and find the image of the human face in the corresponding binarized total image. In con- 
trast to the known approaches, therefore, it is not the intensity value of the individual camera 
pixels which is compared or pixel variations which are looked for. Rather, the invention relies 
on a model structure of the type of a line drawing which may be available, for example, in the 
form of a bit image. 

The model of the face or the "line drawing" is compared with the binarized image by means 
of a modified version of the so-called Hausdorff spacing in numerous scaling, rotation, and 
distortion variants of the image. The application of the Hausdorff spacing to face recognition 
was described, e.g. by B. Takacs and H. Wechsler in "Face Recognition Using Binary Image 
Metrics", 3rd International Conference on Automatic Face and Gesture Recognition, IEEE 
Proceedings, April 1998. Reference is expressly made to that publication and the explanation 
it contains of the Hausdorff spacing. 

The publication mentioned describes the use of the Hausdorff spacing for purposes of face 
recognition. It is stated expressly that face finding is achieved by a method which is totally 
different. It was not taken into consideration in the prior art to apply the Hausdorff spacing for 
the purpose of face finding, among other things because this algorithm takes an awful lot of 
calculating time. 

The fundamental differences between the problems of face finding and face recognition 
should be recalled: Once the face is found in a picture, especially in a moving picture this 
facial picture can be compared with a collection of faces from a data base by almost any 
method of recognition or identification. If the face in question is contained in the data base the 
hit rates in general usually are very high, amounting to approximately 99 %. What is difficult 
with personal recognition, however, is the preceding step, namely first to find the face in any 
random picture and make a perfect "cutout" thereof. What may appear to be a minor differ- 
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1 . A method of finding a face in a binarized image by comparing the dot group of the 
binarized image with the dot group of a face model, the face model being subjected to multi- 
ple two-dimensional transforms in order to locate the face model in the binarized image, and 
the dot groups of the binarizedyimage and of the face model being compared on the basis of 
the Hausdorff spacing between the dots of the dot groups and a position of a face in the bi- 
narized image is found when a measure derived from the Hausdorff spacing fails to reach a 
limit value. 

2. The method as claimed in claim 1, wherein the binarized image is derived from the 
original image by means of edge extraction. 

3. The method as claimed in claim 1 or 2, wherein the binarized image first is compared 
on a small scale with a face model of corresponding small size, the area of the binarized im- 
age in which a face was found is enlarged and compared once again with a face model of cor- 
responding larger size, the enlarging and comparing of the binarized image area and face 
model are repeated, as the case may be, until the face in the binazired image was localized 
with sufficient accuracy. 

4. The method as claimed in claim 3, wherein different face models with different reso- 
lutions are used depending on the size of the binarized image. 

5. The method as claimed in claim 3 or 4, .wherein the edge extraction for deriving the 
binarized image from the original image is carried out with different resolutions depending on 
the size of the binarized image. 

6. The method as claimed in any one of claims 3 to 5, wherein the face model and/or the 
binarized image are rotated and different steps of rotation are used depending on the size of 
the binarized image. 



7. The method as claimed in any one of the preceding claims, wherein the Hausdorff 
measure is determined on the basis of the average value of a certain percentage of the smallest 
minimum Hausdorff spacings, the percentage being between 0 % and 100 %. 



8. A system for implementing the method as claimed in any one of the preceding claims, 
comprising a computing device for calculating the Hausdorff spacing and the Hausdorff 
measure on the basis of the dots of the binarized image and the face model. 
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CQM81MED D6CLARATIOH AND POUER Of ATTORBEY FOR UTILITY PATEMT AWLJCATlW 



As a belou named inventor, I hereby declare: 

That my residence, post office address and citizenship are as stated belou next to my name; 

that I verily believe chat I am the original, first and sole inventor (if only one nam* is listed balou) or a 
joint inventor (if plural inventors are named belwj of the invention which is the subject of an application entitled: 
PERSONAL RECOGNITION rIETHO&jmn SYSTEM, iMCt-UDIWG BODEl-BASEP HtttHUS : said invention beina described and claimed [ ) in 
the attached specification DO in the speeif icatien o* application Serial No, 09/806.785 . filed April L. 2001: that I 
have reviewed and understand the center* of said specification including the claims; that I do not fcnow and do not 
believe the said invention was ever known or used in the United States before my or our invwtior. thereof, or patented or 
described 1n any printed publication in any country before my or our invention thereof or more than one year prior to 
said application; that eaid invention was not in public use or an sale in the United States more than one year prior to 
said application; that said invention has not been patented or made the subject of an inventor's certificate issued 
before the date of said application in any country foreign to the United States on an application filed by me or ay legal 
representatives or assigns more than twalve months prior to said application; that I acknowledge my duty to disclose 
information of which 1 m aware which is material to patentability as defined in 37 CFft 1.56; and that no application for 
patent or inventor's certificate on this invention has been fited in any country foreign to the United States prior to 
said application by *te or my legal representatives or assigns, except as follows: 

COUNTRY/INTERNATIONAL- APPLICATION NUK8SR DATE OF FILING PRIORITY CLAIMED 



GERMANY 


196^7261,7 


(day, month, vesr) under 35 u.S.C. 119/565 
YESJL H0 

5 OCTOBER 199B 


YES SO 



I h«reby claim the benefit under 35 U.S.C, 1119(e) of any united States provisional apphcaiio 
ovisioral Application S.N. • filed 



I hereby appoint L. Lairton Rogers, III, Reg. Ho. 2t,302^D. Joseph Enfllish, Reg. Ho. _ AZ.hHj. M ark C. Comtois, 
Reg. Ho . £6.285: a nd Patrick D. HcPherson, Reg. Mo. 4f,25S to prosecute said application and to transact all business i 
the Patent and Trademark Office connected therewith and to file, prosecute and transact all businets in connection uith 
international applications directed to said invention: 

Address at I correspondence to: ■^Jarjar < -X edy a Fd tJajjbiirp 

1401 Eye Street. P.. SuiteJ OO 
"Ifeifnngton, DC 20QOS 
Address all telephone calls to ( 2023 "558^1515 ~~~~ " — — - — — . 

I hereby declare that all statements made harein of my own knowledge arc true and that all .statements made on 
information and belief are b*liev»cf to be true; and further th« these statements were *ade with the knowledge that 
willful false statements and the like ae made are punishable by fine or i imprisonment, or both, under Section 1001 of 
Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the applicant 
or any patent issued thereon. 



Robert FHschhoU X^^^c^^T ^ f: 

FULL NAHE OF SOLE OR FIRST INVENTOR SIGNATURE wT~~ 

RESIDENCE CIT12EMSHIP 

• ^tz - ^ "'"f^* 1 Coffffunication Systems A S Aw Ifrtterkreuz 19a 91058 Erlanoen. GERMAHY ; ()€^/ L 

POST OFFICE ADDRESS ' . 



